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WHAT IS CLAIMED IS : 

1 . A method for inserting a target image into an image stream of a real 
site taken by a camera, the method comprising: 

receiving a stream of original images taken by a camera of a real site; 
5 for each original image in the image stream, 

identifying a background area within the original image from at least 
one target area in which a target image will be inserted; 

rendering, for each at least one target area, a target image based on at 
least a predefined three-dimensional model of the at least one target area 
10 within the site and the camera's position and pointing direction; 

identifying occlusions within the at least one target area of the original 
image; and 

combining the background of the original image, the at least one target 
image, and the image of the occlusions into an output image. 
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2. The method of claim 1, wherein identifying a background area within 
the original image includes identifying at least one target area within the image based 
on a predefined model of selected surfaces of the site and the camera's position and 
pointing direction. 
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3. The method of claim 2, wherein identifying at least one target area 
within the image comprises: 

receiving camera parameters, the camera parameters including parameters 
indicating the pointing direction of the camera; 
25 rendering the predefined model from the camera's position, based on the 

camera parameters; and 

generating from the rendering a mask for defining the target area within the 

image. 



30 4. The method of claim 3, wherein the camera parameters included 

parameters indicating the camera's position within the site. 
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5. The method of claim 3, wherein the camera's position is predefined 
within the model. 

6. The method of claim 1, wherein rendering, for at least one target area, 
a target image comprises: 

receiving camera parameters, the camera parameters including parameters 
indicating the pointing direction of the camera; and 

rendering the predefined model based at least in part on the camera 
parameters. 

7. The method of claim 6, wherein the camera's position is predefined 
within the model. 

8. The method of claim 6, wherein the camera parameters includes 
parameters indicating the camera's position within the site. 

9. The method of claim 8 wherein the camera parameters are encoded 
onto a video signal generated by the camera. 

10. The method of claim 1 wherein identifying occlusions within the 
image of the target area includes: 

rendering the predefined model of the site with a reference image inserted in 
each of the at least one target areas; and 

comparing the reference image to the target area within original image. 

11. The method of claim 1 further including encoding the output image 
and camera parameters onto an output video signal, the camera parameters including 
parameters indicating the pointing direction of the camera. 
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12. An apparatus for inserting a target image into video of a real site taken 
by a camera, the apparatus comprising: 

a model renderer for generating, a synthetic image based on a predefined 
three-dimensional reference model of a target area within a site from a known 
position of a camera, the synthetic image having a target image inserted in the target 
area; 

an image separator for masking from a video image contained within a frame 
of a video signal generated by the camera, the target area to create a masked 
background image; and 

an image combiner for combining the masked background image with the 
target image. 

13. The apparatus of claim 12 further comprising: 

a second model renderer for generating a second synthetic image based on the 
predefined reference model of the site, the second synthetic image having a reference 
image inserted into the target area; and 

an occlusion separator for comparing the reference image of the second 
synthetic image to the target area of the video image and generating an image of 
occlusions within the target area of the video image; 

wherein the combiner further combines the image of the occlusions with the 
masked background image and the target image. 



